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PREFACE 



President Nixon, on March 3, in his special message 
to Congress on educational reform, called for "a new 
reality in American education," 

This "new reality," he made clear, must focus on 
"the actual amount of lear ning t^^t t^^^^ piapp in a sc hoo 
In measuring educanon ^s effectiveness, ^ he~saia. all oTTl^r 



considerations^'"== ^ faoi -jriTiesV teacher standards, pupil- 
teacher ratios, etc* — are over-shadowed by the answers 
one gets to the simple question: "What do the children 
learn?" American education, he stated, must be based on 
"the outcome of schooling." 

Education geared to instructional outcomes, rather than 
to classic "school characteristics," makes necessary, the 
President noted, "responsible open measurement of how well 
the educational process is working." Further, he added, 
it leads to a system whereby administrators and teachers 
not only are considered responsible for what their students 
learn, but are held accountable for it. 

These elements of the educational reform called for 
by the President are basic to this report, as they were 
to the study which prompted it. 



To achieve educational reform based on measurable 
learning outcomes it undoubtedly will be necessary, as 
the President stated, "to develop broader and more sen- 
sitive measurements of learning than we now have." And 
in this context he noted certain current "immeasurables" 
such as responsibility, wit and humanism. 

Without debating the latter issue, some known 
"measurables" deserve equal mention. For example, a wide 
range of vocational-technical skill areas where results 
most often relate to hands-on performance. Here the 
learning outcomes are, for the most part, self-evident, 
and can be rather easily measured. 

Measurement and validation of learning outcomes 
relating to job skills can only come, realistically, 
from the "user" — the . employer . Only he can provide 
the job entry requirements on which curriculum development 




* 
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should be based. From hiir. we may require the feedback of 
supplementary information to fill gaps in the instructional 
design process* Out of this cross-feeding of information 
will come the documentation — for instructional purposes - 
of specific performance objectives. 

If educational reform centered in measurable learning 
outcomes includes a system within which administrators and 
teachers actually are held accountable for learning out- 
comes, the importance of performance objectives is even 
more enhanced. 

Legally, and ethically, one can be held accountable 
for his actions only if he knows in advance what those 
actions might involve — not merely in broad generalities, 
wide open to interpretation, but in precise terms. A legal 
contract, for example, usually describes in detail what 
performance is required of the parties involved. It 
logically follows that accountability in education must be 
based on pre-determined and specifically prescribed per- 
formance objectices. Administrators and teachers must 
insist on this for their own protection. 



4c 4c 3)e 



Another element of the President's reform message is 
pertinent to this report. While expressing the need for 
a new approach to educational research, the President 
indicated a desire to seek out methods and programs, 
already in existence, which have proved their worth in 
terms of learning outcomes. 

A vast resource of educational programs developed by 
the military services for their own use, but directly 
applicable to civilian education, must be considered in 
this category. 

In the U. S. Air Force, at least, vocational-technical 
training has for years been geared to '^learning outcomes." 
Open measurement of learning results has become routine. 
Constant feedback from the "employer'* (the manpower using 
agencies in the Air Force) is an integral part of the 
educational process, and this makes it possible to* measure 
and validate job entry capabilities. The Air Force 
training establishment is held accountable for regularly 
delivering graduates with acceptable job entry performance 
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to a degree rarely heard of in civilian life. And all 
Air Force courses — covering a wide spectrum of skill 
areas — are based on pre-determined and specified 
performance objectives • Hence, the Air Force has, in 
practice, with a vast backlog of experience, the basic 
elements of the educational reform called for by the 
President . 

A great majority (some estimates: as hi gh as 90%) 
of Air Force career categories involving voc^ational- 
technical skills have clos e ioh entry parallels in 
civilian life* Thus, in exploring Air Force training 
experience we are dealing aot merely with military 
careers but also with civilian job entry requirements • 
Air Force experience in ^'learning outcome" education offers 
model after model of concepts, techniques and materials 
tnat have proved their worth — that are readily adaptable 
to civilian use • If educational reform, as outlined, is 
to come, these models can serve as important money-saving, 
time-saving building blocks. 

Indeed, if educational reform is to take place in a 
tight money market, educational wheels can no longer be 
re-invented, as too often has been the case. Huge research 
investments are being made in educational areas which the 
military has explored and tapped years before. Performance 
objectives are created from scratch at great expense while 
paralleling sets of objectives — tested and validated — 
already are operational in military classrooms. Relatively 
simple adaptations of military-developed course materials, 
rather than the creation of new materials, could save 
countless dollars and research man hours. Investigation 
of the military's resources, as a prerequisite to the 
approval of civilian research projects, would be in the 
national interest. These resources include, not only 
teaching practices and course materials, but, perhaps 
more important, management concepts and techniques developed 
to support instructional systems based on ''learning outcomes. 



* * * 



The study on which this report is based brought these 
conclusions into focus. The report bears down on the 
important first step which must be taken by any school 
system which would practice "learning outcome" education. 

That important first step: coming to grips with what 
it intends to achieve — with performance objectives. 
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During the experiment in Utah, on which this 
report is based, teachers became exposed for the first 
time to courses (Air Force) based on specified per- 
formance objectives. In a sense, they were asked to be 
accountable, in terms of the job entry capabilities of 
their graduates, to the employers they presumably were 
serving. Their reaction to the experience, and their 
action to meet the challenge, is covered in this report. 

If ''learning outcome'!, education is to move forward, 
there is good reason to believe that many thousands of 
teachers will face the same challenge: teaching in a new 
re sult-oriented atmos phere, with their own broad goals 
for student behavioiFTeplaced by specific objectives for 
student performance. And beyond that, the challenge to 
recognize that this new way of life in the classroom 
actually elevates the teacher from the role of instructor 
to the more responsible, more creative, more effective 
role of instructor-manager. 

Our experiment in Utah merely scratched the surface 
of this subject — for it is a broad surface, indeed. But 
scratch it, it did. And the experience may be a start 
toward an understanding of what must come — if true 
educational reform is to be realized. 
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I • SUMMARY 



Background : This report supplements the Final Report 
(Project No. 8-0301, USOE) for the study titled ''The 
Evaluation of Three U, S. Air Force Instructional 
Systems Within Civilian Education," That study examined 
three U. S, Air Force instructional systems (segments 
of electronics, pneudraulics and nurses aide courses) 
in terms of their us'^fulness in the Utah school system, 
and projected their potential usefulness nationally. 
Results were positive in all areas. But during the study 
it became evident that the Air Force materials were ri ot 
hP-ing;^jiff<^H_n ^er Optimal conditio ns, le aving reaso n to 
believe the y co uld be use d even more efxec tivei y , (Tf 
particulay^ importance was T:he^r eT.'a"tlW of specif ic 
performanc e objec tives, as used by the Air F orce, to 
the broader goals of the _conventTonal educator. All this 
gave rise to thls^^speclal report. 

Scope of Report : Basic guidelines are presented for 
introduction of criterion-referenced materials, as 
represented by those of the Air Force, into the civilian 
classroom. The criterion-referenced approach is compared 
with the traditional approach which is followed by 
suggestions for selection, preparation for use and use 
of criterion-referenced materials. 

Recommendation : That practical g uidelines for teachers be 
formulated to implem ent crite rigrr^ref erenced courses in 
vocational-sRtn r^area s , drawing fully on procedures 
developed and dTSXTecfed by the U, S. Air Force, and on 
other sources — such guidelines to provide step-by-step 
instructions for classroom use. 
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GUIDELINES FOR INTRODUCING 
LEARNING OUTCOME INSTRUCTION 



by 



James H. Straubel 
Aerospace Education Foundation 



II. INTRODUCTION 



The U. S. Air Force, in recent years, has mounted a 
massive and pioneering effort in developing instructional 
systems, including some 5000 hours of programmed materials, 
with each course designed to achieve at least the outcomes 
that are specified in advance. 

Sensing the potential usefulness of this instruction 
in civilian schools, the Bureau of Research, U, S, Office 
of Education funded a project to permit the Utah Department 
of Vocational Education and the Aerospace Education Founda- 
tion to explore this potential. 

(See Final Report, Project No. 8-0301, Grant No. 

OEG-3-8-.080301-0035 (085), "The Evaluation of Three U. S. 

Air Force Instructional Systems Within Civilian Education,'' 
December 31, 1969) . 

The resulting 18-months study addressed itself primarily 
to these two basic questions: 

1. Is it feasible to use Air Force instructional 
materials in civilian schools in the state of 
Utah? 

2. If it appears feasible in Utah, how likely is it 
that it will be as feasible elsewhere? 

In the study, Utah instructors offered segments 
of three different Air Force courses (which they selected) , 
using procedures as similar as possible to those set out 
by the Air Force. Student performance in these segments 
was then compared with that of students receiving convention- 
al instruction. Information also was gathered on the 
modifications required of Air Force materials, and on the 
reactions of students and instructors to such materials. 
Used in the study wei*e: 

1. A 90-hour segment .from the Electronic Principles 
course ; 
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2. A 60-hour segment from the Aircraft Pneudraulics 
course ; 



3. A 20-hour segment from the Medical Service 
Specialist course. 

The electronic segmenx was taught at five schools, 
each of the other two segments at a single school. 

As indicated in the Final Report, the results were 
highly encouraging. Students using Air Force procedures 
and materials performed at least as well as those ta king 
the standard course, or significantly better, and generally 
in less'^^^ ime . "The amount of modification required to adapt 
the Air Force materials to civaTiirr use was not excessive. 
Perhaps even more significantly, insTfirctors have-^ontinued 
to use the Aix_JFprce niaterials in preference to their 
own, and the Utah DepaTrFment of Vocational Education has 
proposed for a doption o n a state-wide basis the objectives 
of the electronics c ou fs eHse gme n 



The success of the Utah study seems to confirm the 
belief that Air Force ^ourse m atexiaJ-A used success- 

fully in a civilian environment. More thaif tTiat, since 
the materials^were^hot used under optimal conditions, 
there is good reason to believe that their use can be made 
even more effective than they were during the study. This 
prediction is based upon identification by investigators 
of a number of factors that probably depressed both the 
potential effectiveness of the Air Force materials (as 
defined by student achievement) and their efficiency (as 
measured by performance gain per unit time) . It did not 
seem appropriate to explore these factors in depth in the 
report of the main study, but since they bear on the question 
of how to make the most effective use of Air Force materials, 
they are described herein. 

This report is the result of many spin-offs from the 
main study, as it progressed; it is based on several 
f ollow -on_workshops with instructors engaged in the Utah 
project"^ on personal Interviews-'With Utah teachers and 
administrators, and numerous meetings with Air Force 
training specialists. 

This report offers basic guidelines for the intro- 
duction of criterion-referenced materials, as represented 
by those of the Air Force, into the civilian classroom. 
These guidelines are based on the experience gained in 
the recently completed research project in Utah, referred 
to earlier in this report, on interviews with Utah 
administrators and instructors and on other related 
experiences. The report begins by describing the 
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criterion-referenced Air Force materials and comparing 
and contrasting them with conventional instruction 
materials in an attempt to furnish the reader some 
perspective on this matter. The report then offers 
suggestions on the following: selection of the materials, 
preparation for use and actual use of criterion-referenced 
materials giving consideration to teacher attitudes and 
teacher orientation. 




III. AIR FORCE MATERIALS: AN OVERVIEW 



Description 

The expression "Air Force materials" as used in the 
present context, needs some clarification. The many 
charts, drawings, mock-ups, and simulators developed 
as training aids for the Air Force instructor are not 
necessarily a part of the materials referred to here. 

The materials under discussion are criterion- 
referenced. That is, they are developed to help a rela- 
tively specific target population atta in pre-speci f ied 
criteria of pelr f or manc e When used accDrding"^to prescribed 
conditions. During development, the materials are tested 
for effectiveness and amended as necessary until they 
perform satisfactorily. When they are used as prescribed, 
there is a high probability that the desired results will 
be achieved. 

Thus — and this is the important distinction — 
the Air Force materials are not simply aids used to 
supplement the efforts of an instructor. Instead, the 
materials themselves are the essence of the course. 
Plainly, this changes the role of the instructor. Instead 
of being the major source of information, as in a conven- 
tional course, the instructor has the no less important 
or less difficult tasks of managing instruction (seeing 
that the right materials reach the right students at the 
right^ time) and of diagnosis and remediation (finding 
the problems of individual students and taking appropriate 
steps to solve them) . With the instructor spending less 
time tran ^itt ing information, he has niorfi_iime for 
d e c i s i o n-making, o n jLns3brAac4;ip nal procedure s, and more time 
to serve as a consultant to students who deserve personal 
attention. Thus, he is more intimately aware of individual 
needs. And with classroom progress controlled by student 
performance, rather than time-controlled, the instructor 
holds a tighter rein over what happens in the classroom. 

The nature of the materials varies. They include 
vide Ota p^s^antrifr(n±on-^pic^ film s , progra mmed te xtbooks , 
workbooks, and instructionaT devices . ^ ' ^ 



Development 

The development of these materials differs considerably 
from the development of materials intended simply to assist 
an instructor or to provide practice. In the first place, 
before instructional development begins, a serious attempt 
is made to identify the performance expected of the 
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proficient student. These objectives describe the 
end-product of instruction, the performance expected of 
the proficient student. Where appropriate, they also 
describe the conditions under which the performance will 
be expected to occur and the criterion of acceptable 
performance. The objectives thus provide the basis for 
deciding what to teach and how to tell when teaching has 
been successful. 

Once objectives have been defined, a test is designed 
for determining whether the objectives have been achieved. 
This instrument, typically referred to as a criterion 
test, includes whatever items are relevant for assessing 
achievement of each objective. Since the criterion test 
is designed to determine wh ether an objective has been 
achieved, questions usually asked during the development 
of test items — whether the item has a particular level 
of difficulty or whether the responses of a group of 
students are normally distributed — become irrelevant. 
What is relevant is only whether the test assesses the 
intent of the objective and whether it is clear to the 
student — no more and no less. 

With the objectives and a test instrument available, 
instruction is prepared in draft form and then, using 
members of the target population, the instruction is 
tested and xcvised as many times as necessary to ensure 
the desired outcomes . 

Classroom Use 

When such criterion-referenced materials are used, 
a rather different philosophy prevails than in the tra- 
ditional classroom. In essence, the instructor becomes 
the ally oX._U]je_stujd^ helping him atrtl lWe^he -objectives 
of the course, regjirjdles's'^'islrw^^ objectives. 
Ideally, of course, each student w6iQ*3'^ pr ocefel^ at his own 
pace, taking as long as he needed to complete a segment 
of instruction. With a few exceptions, the Air Force 
procedure does not go quite this far. Typically, the 
practice is to use a group-paced technique wherein all 
students are exposed to the material at the same time. 
Even so, the object is still to instruct so that the criteria 
are attained by as many students as possible. 

In summary. Air Force materials of the type under 
discussion are developed according to criterion-referenced 
procedures calling for development and testing of materials 
until they result in the student performance specifiied 
in the objectives. 
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IV. TRADITIONAL INSTRUCTION 



Most instruction, whether military or civilian, 
inarches to a different drummer. Though an experienced 
instructor may have his broad goals fairly well in mind, 
he tends not to translate those goals into objectives 
which specify his intended outcomes in terms of student 
performance. Hence he cannot design his instruction to 
achieve those outcomes effectively and consistently. 

The usual procedure is for the instructor to prepare 
lesson plans outlining the content he wishes to cover 
and identifying the instructional aids he intends to use. 
Because schools seem designed more for administrative 
convenience than for maximizing student learning, the 
instructor is constrained to instruct for fixed periods 
of time. He lectures, demonstrates, discusses, or asks 
students to complete laboratory exercises. Periodically, 
he administers an examination in which, using a sampling 
technique, he tries to determine how much of what he has 
taught has been learned by the students. These examina- 
tions are scored and grades are assigned on a norm-referenced 
basis, that is, a few of the highest scoring students are 
assigned the highest grade, the next highest scoring 
group of students is assigned the next grade, and so on. 
Though such a scoring system reveals which students happen 
to perform better than their chance neighbors, it reveals 
little about whether any student can perform as desired. 
Though the system can rank students on the. basis of their 
ability to respond appropriately to the items, it reveals 
nothing about the number of students who can perform 
according to a standard of acceptable performance. 

Typically, the instructor in such a system works 
from textbooks and lesson plans . He measures his perfor- 
mance in terms of content covered and hence feels no need 
to have his intended outcomes written down in the form of 
objectives. Because he does not have measurable objec- 
tives, he cannot have a criterion examination based on 
objectives. He could, if pressed, describe the essential 
characteristics of his students as they influence his 
instruction and their ability to learn, but probably has 
never committed this information to writing. He cannot 
tell how effective his instruction is since this requires 
the administering of a criterion examination both before 
and after instruction. This instructor tends to think more 
in terms of instructional procedures (what he intends to do 
to his students) rather than in terms of instructional 
outcomes (what he intends for the students to be able to 
do as a result of his instruction) . 
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V. THE UTAH EXPERIENCE 



Initially, in the Utah study, the instructors involved 
apparently felt that because they were experienced at 
instructing in a conventional setting they understood the 
Air Force instructional objectives with which they were 
working. Further, they tended to behave as though the 
content headings in their lesson plans were the same as 
instructional objectives. As a result, they compared 
the specific statements of objectives associated with 
Air Force materials with items of content included in their 
own courses, a comparison not of outcome with outcome 
but of outcome with procedure. It was only while the 
course was in progress that the instructors came to realize 
that outcome statements and process statements differ 
substantially and that they had not fully appreciated 
the implications of Air Force objectives. 

On completion of the experiment, most instructors 
interviewed urged that if Air Force materials are again 
made available, steps should be taken to insure that 
instructors understand the nature of an instructional 
objective and its implications. One or two instructors 
went so far as to suggest that such materials should be 
made available only to instructors who have written 
measurable objectives on their courses with which to 
appraise the relevance of the new materials. They pointed 
out that if an instructor relies on these materials to do 
the entire teaching job when, in fact, the objectives of 
the materials are not the same as his, he will be dis- 
appointed in the outcome and be inclined to conclude that 
the new materials and procedures are not satisfactory. 

Out of workshop sessions and personal interviews 
came the definite conclusion that, in designing for most 
effective use of Air Force materials, special attention 
must be given to instructional objectives. Conditions 
must be arranged so that instructors considering the use 
of Air Force courses will understand the extent and 
limitations of these materials. 

An instructor who is not knowledgeable about the 
implications of objectives is hard pressed to make sound 
decisions about procedure. For example, when he is 
confused about the difference between means (content) and 
ends (objectives), he tends to make decisions on the basis 
of a content comparison; that is, he will tend to conclude 
^that Air Force materials are somehow lacking if they do 
not cover the same content as his own course. Often, 
less or different content is needed to achieve a given 



16 



objective than is used by an instructor, but unless he 
recognizes this he may find Air Force materials wanting. 
In the Utah study, for example, instructors of the elec- 
tronics course concluded that the sub-course in Direct 
Current developed by the Air Force matched their own 
intent except in the area of mathematics. It was their 
judgment that they had to teach more mathematics than 
was taught in the Air Force sub-course to reach their 
Direct Current objectives. For this reason, they decided 
to add mathematics instruction to the Direct Current 
sub-course . 

In fact, students completing the unaugmented Air 
Force course performed significantly better than those 
completing the regular course, despite their lack of the 
additional mathematics. It is clear that the added mathe- 
matics was not relevant to achievement of the objectives 
for the Air Force Direct Current sub-course. In a workshop 
session following the Utah study, the electronics instructors 
acknowledged that the additional mathematics was relevant 
to achievement only in the Alternating Current sub-course 
to which students were assigned following the Direct 
Current segment. Traditionally, a mathematics sub-course 
is taught in a block before electronics theory, even 
though this mathematics segment is not required for the 
sub-course that immediately follows it. 

Objectives 

Conflicts in objectives also were apparent in the 
aircraft maint.e^nance portion of" the experiment. Initially, 
Utah teacfiers select eg7"th;e_T Vheels , Tires and Bi;akes 
segment of the Air FoVce'^Aircraf t Mechanics course taught 
at Sheppard Air Force Base, Texas. A number of" problems 
arose, the major one being the lack of training equipment 
in Utah comparable to Air FoiT^^^equTpment useSr TiTTe aching 
the course. Another: high adaptation costs. Therefore, 
the Pneudraulics Repairmen course at Chanute Air Force 
Base was offered as a reasonable alternative. This 
course was accepted, although only one-third of the equip- 
ment specified for the course could be obtained from surplus 
sources. Equipment problems like these are beyond control. 
Other problems are not. For example, without precise 
objectives to guide them, the Utah teachers estimated 
that the scope of their own pneudraulics course was some- 
where between the 3-level and 7-level of the Air Force 
course. Therefore, extra effort was required to inter- 
weave the lessong^ plans for the two levels. This also 
affected the interweaving of the tests at these levels. 
Then, materials from the Air University had to be added 
to cover information not found in the course for pneu- 
draulics repairmen. 
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Only after the course had been deiSigned and placed 
in the school system was it determined that a different 
level Air Force course would have been more effective. 
In retrospect: had the course objectives been spelled 
out clearly, an Air Force officers' course for pneu- 
draulics' supervisors, rather than an enlisted men's 
course for pneudraulics repairmen, would have been 
offered the Utah instructors, and would have met the 
requirements • 

Yet, a comparison of objectives is only a part of 
course selection. Even where there is congruence between 
civilian and Air Force sets of objectives, it may be 
that more equipment or more course modification is 
required than is practical. 

Equipment Needs 

The equipment problem was revealed early in the 
feasibility study when Utah instructors were reviewing 
potential Air Force courses. One course was rejected 
specifically because theu^g ost of equipment a ssociated 
with the instructAon-^samplv ^could not j pe^ borng''lJy the 
school. Fortunately, this appears to be an issue that 
should pose no particular problem if a detailed equipment 
list is attached to every Air Force sub-course under 
consideration. A school can then, readily compare the 
required materials with a list of what it can make avail- 
able. 



Modification 



The question of modification is less simple. In the 
case of the electronics sub-course selected for the Utah 
experiment, no modifications of the content or structure 
of the materials were considered necessary. Because 
closed-circuit television was not available in the Salt 
Lake City schools, however, the Air Force videotapes 
associated with the Dirg c^t Curr ent sub-course had to be 
converted into motion p ic t u r e t rlrm — ThtJ- uOaL^to produce 
one film copy of^each Videotape, whileT^iTffe high, was 
not indicative of the unit cost with a reasonable volume 
requirement . 

Modification may have to be made in the content of 
the materials. Generally, as with the medical specialist 
course, instructors can be expected to decide that 
certain military terminology should be modified to civilian 
terminology. Another example of a potential materials 
modification was provided by the electronics sub-course. 
Instructors reported that the films provided overly long 




pauses for students to write responses in their guides, 
They suggested that substantially shorter pauses would 
have made the films more interesting without damaging 
their effectiveness. 
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VI. SELECTION OF THE MATERIALS 



Effective use of instructional materials must be 
preceded by effective selection. 

In the process of selection, these basic questions 
should be asked: 

1. Will these materials do all or some part of what 
I want to accomplish? 

2. Is it feasible to use them as directed by the 
developer? 

3. How effective are they if used according to 
directions? 

4. How must these materials be modified to make 
them useful or workable in my instructional en- 
vironment? Considering the time, money, and 
facilities needed, is modification feasible? 

Given the appropriate tools, answers to these 
questions are not difficult to determine. The problem is 
the absence of the "appropriate tools." And their absence 
is the heart of the problem of selection. There are some 
conditions that must be met from both sides of the fence. 

First, the teacher must have written out: 

1. Objectives describing in measurable terms the 
performance expected of a proficient student; 

2. A description of the target population (available 
students) ; 

3. A measure of how effective his own instruction 
is in achieving his objectives. 

(The last can be obtained only if he also has a 
criterion test which is based completely and exclusively 
on his objectives.) 

Then the developer of the instructional materials 
must provide similar information so that a comparison can 
be made. 

To the degree that this information is lacking on the 
part of either the instructor or the developer, the 
decision about the utility of the materials in the new 
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setting becomes more difficult. Good selection is 
dependent upon both instructor and developer providing 
precise information about objectives, target population, 
and instructional effectiveness. Implicit in that 
statement is a large part of the answer to the question 
of how to make the most effective use of any new materials. 

Unfortunately, it is seldom that all of the informa- 
tion needed for good selection is available. Instructors 
and developers are equally culpable in this. And since 
the situation is not one to be remedied overnight, the 
problem is, as usual, one of determining how to make 
the best use of an imperfect world. 

To select instructional materials intelligently 
one must first have instructional objectives which state 
explicitly the kinds of desired student performance. 
Given objectives, the instructor should be able to select 
instructional procedures and content that will facilitate 
achievement of those objectives. Without such objectives, 
he can only operate in a fog of his own making. 

A school considering the use of criterion-referenced 
Air Force materials must initially ask itself, "How well 
are the objectives spelled out for the course as it is 
currently taught?" If the objectives are written in 
performance terms and if those objectives are based on 
some form of task analysis and projection to insure their 
importance, then it is reasonable to move to the next 
step, that of selection. 

If, on the other hand, the instructional intents are 
not well defined — that is, if instructors protest that 
"We know what we want to teach" but cannot produce a 
written statement of the desired outcomes — then additional 
steps must be taken before potential materials are reviewed. 
Exactly which steps are indicated depends primarily upon 
the way an institution makes decisions about curriculum. 
If individual instructors are allowed to make all decisions 
about the outcomes of instruction, then all instructors 
must be schooled in procedures such as task analysis and 
the preparation of instructional objectives. If, on the 
other hand, a team of specialists has responsibility for 
identifying the desired outcomes, then a different and 
simpler route can be followed. 

The Utah experience is instructive in this regard. 
It became apparent that the instructors charged with 
reviewing Air Force materials could not, in most cases, 
say precisely what performance outcomes they wanted of 
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their students, and further, they could not say for sure 
what skills were relevant to the kinds of jobs likely to 
be available to the young men they were training. 

It would be a mistake to conclude that the Utah 
instructors were in some way less competent than their peers, 
however. Instructors tend to teach as they themselves 
were taught and in a fashion that is acceptable to their 
parent institutions and to the educational community. 
Since teachers are no more exempt from the laws of behavior 
than anyone else, they tend to do those things that lead 
to positive consequences and avoid doing those things 
that lead to no consequences or to ave^rsive consequences. 

In the educational system as currently operated, 
it is clear that the consequences related to effective 
instruction are usually few. The system simply does not 
revere results that are defined in terms of student 
performance. Generally, the rewards of the system 
(salary, promotion, status) are dispensed not for effec- 
tive instruction but for the number of months served and 
for the number of credits earned. Thus, the instructor 
who has served longest with an institution tends to be 
highest on the salary scale, regardless of how successful 
he is in facilitating desired student performance. In 
such a situation it is not reasonable to expect instructors 
to behave much differently than they do. 

Before teachers will accept the idea of student 
performance as the primary object of instruction, there 
must be a climate of acceptance for such a viewpoint. 
Specifically this means that the administrators must be 
favorably disposed toward it, capable of identifying 
teachers who are behaving appropriately from those who 
are not, and willing to make favorable consequences 
contingent upon such activity. The administrator must 
revere an instructor's output (as measured by the extent 
and direction of student performance) more than he reveres 
what is taking place in the instructor's classroom. The 
administrator who is unaware of the criterion-referenced 
approach to instruction is, through word and deed, likely 
to punish instructors inadvertently for trying to use it, 
thus influencing the teacher to abandon his attempt to 
change drastically his mode of instruction. 

Experience indicates that it is not enough for an 
administrator merely to be aware of and open-minded about 
the criterion-referenced approach. If he is to influence 
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his faculty in that direction, it is imperative that he 
demonstrates active enthusiasm in the new approach, that 
he motivates his staff to shift their frame of reference, 
and make it clear that he will sustain them in the many 
setbacks and pitfalls that lie ahead. Those who conduct 
teacher training workshops in the preparation and use of 
instructional objectives report that it is not difficult 
to convince the teacher of the logic and humanity of the 
approach. And, within a day or two, it is possible to 
impart enough skill and motivation to actually influence 
the teacher to try to write objectives • But implementation 
of the criterion-referenced approach to instruction calls 
for more than a brief in-service workshop. It takes 
time to develop the habits of thinking associated with 
the new approach. 

This discussion is not intended to imply that 
teachers ahd administrators need inordinate preparation 
before they can effectively use criterion-referenced 
materials such as those developed by the Air Force, or 
that these new materials would be of no value to instructors 
teaching in the traditional manner. The point is that 
the true effectiveness of such materials is dependent 
upon an instructor who understands and applies the criterion- 
referenced approach. 

Now let's review the four basic questions we said 
should be asked in the selection of criterion-referenced 
materials . 

1* Will these materials do all or some part of what the 
instructor wants to accomplish? 

The basic procedure for answering the question is to 
compare, item by item, the objectives of the desired 
course and those of the proposed material. Objectives 
are specified for each validated course prepared by the 
Air Force and if an instructor has objectives for his own 
course, regardless of who prepared them, a comparison is 
relatively straightforward. The degree of overlap can be 
readily determined and if objectives are rank ordered in 
terms of importance even more information can be developed 
regarding the relevance of the proposed materials. 

On the other hand, if Air Force materials are used 
at institutions where course objectives are not stated in 
a manner which will allow comparison, it is likely that 
instructors will compare lesson plan content with Air 
Force outcome statements, and make less than optimal 
decisions. As pointed out by the Utah instructors, a 
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teacher who selects a criterion-referenced course will 
tend to rely on it to do the bulk of the teaching job. 
When he discovers that it does not in fact do the entire 
job (even though no one claimed it could do so) he, is 
likely to blame the materials for his failure* More 
important, he is likely to become. not only disenchanted 
with but confused about criterion-referenced instruction 
itself. 

2 . Is it feasible to use the materials as directed by 
the developer? ' 

Though new materials may be appropriate and feasible 
from the standpoint of cost and ease of modification, 
they may still be unusable by a particular school. In 
the Air Force electronics sub-course tested in Utah, for 
example, the recommended procedure is to use a series of 
motion pictures as the primary means for explaining the 
subject material. As he watches a film, the student is 
expected to write responses to questions in a workbook. 
The instructor's prime functions are to ensure that the 
films are seen at the proper times, to answer questions, 
and to help students. He is no longer a lecturer; instead, 
he is a manager and a consultant. If a particular 
institution does not allow an instructor to operate in 
this manner,. then validated courses like those of the 
Air Force are again contraindicated . 

To determine whether the recommended practices are 
feasible : 

1. Identify the procedures recommended by the 
developer. (If they have not been described 
explicitly, it will be necessary to observe 
the materials being used, or confer with the 
developer) . 

2. Have potential instructors indicate whether they 
are willing to follow the recommended procedures. 

3. Have administrators indicate whether they are 
willing for their instructors to follow the 
recommended procedures. 

3 . How effective are these materials if used according 
to directions? 

It has to be stressed here that the effectiveness 
of any Instructional material cannot be assessed merely 
by inspection or by analysis of the structure or content. 

If Air Force materials are accompanied by validation 
data, then their potential effectiveness in a new en- 
vironment is predictable, provided the target populations 
match and provided the materials are used according to 
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directions. If validation data are not available or if 
the conditions regarding target population or proper use 
cannot be met, then the only reliable way to determine 
effectiveness is by actually trying them out on a sample 
of one's own target population. 

Questions about how much better the new materials 
might be than traditional materials are probably unan- 
swerable, given the procedures employed in most schools. 
An answer requires comparative data about the effectiveness 
of both the existing course and the validated materials. 
If such information is available for the latter, but not 
for the former, as generally is the case, the question 
''What can be expected in the way of improved student 
performance?" makes little sense in this context and 
should be deleted in any consideration of new materials. 

4 . How must these materials be modified to make them 

useful or workable in the new instructional environ- 
ment? Is modification feasible? 

Even though a set of materials may be completely 
appropriate in that the objectives match those of the 
instructor, it may well be that it is not feasible to 
use those materials. They may need too much equipment 
or space, for example, or perhaps it would take too much 
effort to modify them. This would clearly be the case 
if, say, the school had to supply large and complicated 
mock-ups or simulators, or if costly modifications were 
required . 

Modification and Cost Feasibility 

Based on experience with the Utah project, the 
procedure recommended for determining modification and 
cost feasibility is as follows: 

A. Having identified one or more sub-courses as the 
result of an objectives' comparison, identify 
the equipment and materials required for instruc- 
tion associated with the objectives. 

B. Calculate the cost of required equipment and 
materials not currently available at the school. 

C. If additional equipment is required, determine 
whether suitable space and power are, or can 
be made, available. 

D. Review the materials and identify concepts and 
terminology considered inappropriate for school 
use . 
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E. Assess the importance of deleting inappropriate 
concepts and terminology. This can be most 
handily accomplished by asking, ''What is the 
probable consequence of leaving them where they 
are?'' 

F, For concepts and terminology judged detrimental, 
estimate the cost of deleting them. 

Only if both' instructors and administrators are 
willing to carry out the instruction as designed can the 
use of the materials be considered feasible. If either 
instructors or administrators are reluctant, the materials 
should not be used, for they are sure to fail in such 
circumstances . 

In the real world there are, of course, degrees of 
feasibility. The issues are not as clear-cut as this 
report may make them sound. The practice of instruction 
is still far from being a science, even when the methods 
used in the preparation of criterion-referenced materials 
have removed some of the margin for doubt. This being 
the case, it is relevant to speculate as to how much one 
can deviate from recommended procedures in criterion- 
referenced material before results begin to be degraded. 

More specifically, is it not possible to use Air 
Force materials in ways different from those recommended 
and still get good results? The answer, of course, depends 
on what is meant by "good results." As was seen in the 
main study, the procedures followed only approximated those 
recommended by the materials' developers. Even so, students 
using these materials performed at least as well or better 
than those exposed to the traditional instruction. As 
can be seen from the post-test results, however, to 
attain performance that is as good as traditional instruc- 
tion is hardly an achievement. Criterion-referenced 
courses, as developed by the Air Force, were designed on 
this premise: That unless a high percentage of students 
accepted for a course (80 per cent or more) perform as 
prescribed by the objectives, the instruction must be 
counted a failure. If instruction has been validated for 
a particular target population, and if members of that 
target population have been admitted to the course and 
non-members selected out, instruction should achieve 
whatever results were achieved during field testing of the 
materials . 



VII. PREPARATIONS FOR USE 



Most of the preparation steps have been implied 
in the previous section of this paper and do not require 
much more in the way of explanation. If additional 
equipment is required that equipment must be procured and 
available before the course begins. If materials must 
be modified, the indicated modification should be completed 
during the preparation stage. 

In addition to the obvious preparation steps, a few 
recommendations are in order with regard to instructor 
preparation : 

1. Instructors should be briefed on the extent and 
limitations of the Air Force objectives as they 
compare with those of the course to be taught . 
They should be reminded immediately before the 
course begins about any content and activities 
to be added to meet the course objectives. They 
also need to be reminded about which objectives 
the materials are likely to achieve without much 
help. 

2. Instructors should be briefed on the essential 
procedures to be followed. The preferred technique 
is for both the instructors and their adminis- 
trators to review the procedures and the ways 

in which it is possible to tell whether the 
procedures are being followed. This makes it 
more likely that administrators will not in- 
advertently penalize instructors for carrying 
out procedures which may be foreign to the 
school but which are appropriate for the materials 
being used. 

3. Probably the most important step is to review 

the procedures to be used for evaluating students. 
It is in this area that the largest discrepancy 
exists between current practice and criterion- 
referenced practice. As previously mentioned, 
standard practice is to give examinations con- 
sisting of items which sample coverage of the 
materials taught, assigning grades on the basis 
of how well the individual student performs in 
relation to his happenstance neighbor. Evalua- 
tion in a criterion-referenced course is carried 
out differently. The examination is strictly 
keyed to the objectives and grades are assigned 
(if assigned at all) on the basis of how well 
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the individual achieved the assigned objectives. 
With such a system it is quite possible for most 
or all of the students to achieve as desired, 
in which case it is mandatory that they receive 
the highest grade. If the administration of an 
institution is not willing for students to be 
evaluated for achievement on these terms, this 
fact should be well understood before criterion- 
referenced instruction is undertaken. 
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VIII. USE OF THE MATERIALS 



Flexibility is the key to using criterion-referenced 
materials. Unlike a traditional course, in which time is 
fixed and the amount of learning is variable, a criterion- 
referenced course continues as long as necessary to achieve 
the desired outcomes — in other words, the amount of 
learning is fixed and time is variable. To operate 
successfully, the instructor must keep his eye on the 
objectives. He must remember that his function is to 
help the student achieve the objectives, using the 
materials available . 

Instructor Guidance 

To increase the probability that materials are used 
under favorable conditions it is recommended that 
instructors have periodic access to those more experienced 
with the procedures. The effectiveness of consultants 
has been established in other projects where instructors 
have been expected to introduce novel instructional tech- 
niques. For example, during the early implementation of 
Project PLAN, a comprehensive individualized instructional 
system for grades 1-12, it was clear to the developers 
that a one-week workshop in the individualized procedures 
was not sufficient to cause teachers to implement the 
system as designed. Frequent, regular visits from a field 
consultant, however, rapidly increased the teachers* 
ability to solve minor problems and to perform as desired. 
In Project READ, a programmed course in reading for ghetto 
youngsters, a large part of its success is attributed to 
the consultants anS aides made available to the teacher 
using the new materials. 

Organization 

A number of procedural matters which have some bearing 
on the use of Air Force materials came to light during 
the Utah study. For one thing, instructors pointed out 
the importance of 'V^rganj Ti^X^ ^ > " emp hasizing^Jbhe need 
to have all material^ jitt-4;he*^gh4H^ fou g J l P ^^^^Lh e righ t time 
and to have ^'asiSrooms available on schedule. If'ov those 
unfamiliar with the school_scene t^ p fiay se^^yi be a 
point too ^^43a^i3ai»-''^rTri^ . It is not a triviar problem, 

however. n,j?.ublic schoflJjSL».^t» ^ A .to juggle 

^the assignment of class rooms during a semest er. While 
this may be^^S^mftMr-proFlem for an instTUcfTor who has 
lierely to move a textbook and a piece of chalk from one 
room to another, it poses a severe problem for the 
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instructor who must move a great deal of material and 
equipment. Instructors participating in the Utah 
experiment recommended that Air Force materials be used 
only by institutions willijig^jtojna^ flgn_ coinmitments 
for s pace and f acilitiesT^As they pointed out, a lapse . 
in these matters is far more visible and upsetting to the 
student when the course is highly organized and carefully 
sequenced, as is the case with the Air Force courses. 

Instructor Attitudes 

Another issue in this general category concerns the 
attitude of instructors. It is well known that a student's 
attitude is significantly influenced by the attitude of 
his instructor; if the instructor models apathy or 
dislike toward his subject matter or the materials he is 
using, it is likely that his students wiil adopt a similar 
stance. The Utah instructors were sensitive to the impor- 
tance of the instructor being favorably disposed toward 
his topic and his procedures. They recommended that 
administrators be urged to refrain from requiring an 
instructor to use Air Force materials if he is not disposed 
to do so. 

Indeed, at the outset of the experiment, several Utah 
instructors were highly skeptical of the prospects of intro- 
ducing Air Force materials into their classrooms — not 
from an anti-military standpoint (the Air Force is the 
largest employer in the state) but with regard to the new 
instructional approaches ixivolved, and the forthcoming 
evaluation of old vs. new. On this latter point, par- 
ticularly, the skepticism could well be understood. In 
each subject area under study, the instructors involved 
had played major roles in the development of the con- 
ventional course. Air Force techniques and course materials 
were, in effect, being placed in competition with "my course. 
For this reason alone, adaptations to Air Force materials 
were encouraged, and ultimately instructors found personal 
identification with the Air Force course segments. 

A good example of this trend was evident in the 
electronics portion of the experiment. The conventional 
course was the traditional lecture course with some 
discussion sessions, and with no visual aids. The Air 
Force course was entirely film-based, supported by student 
study guides and workbooks. During pauses in the film 
showings, students were expected to answer questions in 
their workbooks. 

Some instructors, apparently feeling they had been 
replaced by strips of celluloid, were inclined to open a 
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class session with a remark such as: "Well, here's some 
Mickey Mouse for you, but that's what the program calls 
for/* Yet, by the end of the experiment, the electronics 
teachers, accepting the visual-oriented Air Force course 
as superior to their own, acted and spoke like instructor- 
managers as they offered these recommendations for 
implementing f i Im-based instruction : 

1. To increase the probability that students will 
follow instructions and make active responses 
as they watch filmed sequences, instruct them 
to fill out the response books and tell them 
that you will be looking at their books. 

2. Use the film to introduce a new topic rather than 
to review it. Introduce the film yourself before 
showing it. 

3. If you are working in a room that is quite dark 
when the shades are drawn and the lights are out 
for screening a film, a single spotlight aimed 
at a corner of the room will provide enough light 
for students to write in their TVI Guides. 

4. Stop the film occasionally to change the pace 
and to augment explanations where you feel this 
may be helpful. 

5. At the beginning of the course, tell the students 
how and why the films were prepared. This brief 
procedure should increase the student's acceptance 
of the films. 

Utah instructors in all three subject areas under 
study agreed in the superiority of the Air Force course 
segments, to the extent that, at their initiative, these 
segments have become part of the regular curriculum. 

In this connection, the Chief Nurse in charge of the 
nurse's aide course deserves special mention. At the 
outset, she was perhaps the most outspoken of the instruc- 
I tors in questioning the use of Air Force materials. At 
l\ the end of the experiment she not only favored these 
i materials, but spent her own time, after school hours, 
^1 doing adaptations so the Air Force course segment could be 
\i integrated into the regular curriculum in time to meet 
|;scheduling deadlines for the following semester. In so 
doing, she was making an extra effort to replace a course 
Vunit she personally had developed. 

I 
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It is pertinent to note, in assessing these teacher 
attitudes, that the quality of materials was not necessari 
a determining factor. In fact, student surveys revealed 
(and teacher interviews confirmed) that the Air Force 
films used were not judged to be high in quality of 
content; some students said some films were ''boring/' 
Yet, both students and teachers favored the film-based 
instruction over conventional lecture-discussion sessions. 

The use of visual aids in itself (quite apart from 
film quality considerations) was regarded as a positive 
factor. But more important, both teachers and students 
seemed to be reacting favorably to a new way of life in 
the classroom — even though they were getting only a 
taste of what that way of life was meant to be under [ 
the "learning outcome" instruction system which produced 
it. 
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IX. AN APPRAISAL ~ FOR FUTURES 



In moving from conventional to criterion-referenced 
curricula, teacher attitude probably is the single most 
important factor in measuring success or failure of the en- 
deavor . 

Additional to the teacher reactions already covered, 
and as noted in the basic report, this teacher-centered 
activity has taken place in Utah: 

* Many project teachers voluntarily updated themselves 
with literature on the application of performance objectives 
to classroom procedure. 

* Midway in the experiment, selected teachers par- 
ticipated in a workshop, sponsored by project investigators, 
which confirmed the need for closer school-employer 
relationship to more precisely establish job-entry 
requirements . 

* * To fill this gap, selected teachers participated 
in a two-week workshop with union and industry representa- 
tives to establish these requirements and relate them to 
classroom experience. This was the first step toward 
development of an "articulated curriculum" linking high 
school and post-secondary electronic courses in the state, 

* Selected teachers participated in a summer work- 
shop (1969) to help prepare, for the above effort, a 
compendium titled "Articulated Guide for Secondary and 
Post-Secondary Vocational Electronics." (Copy of Guide 
included in Appendix) . 

* Teachers throughout the state, in every school 
offering electronics courses, have participated in the 
evaluation of the Guide — which essentially is a 
compendium of the performance objectives for the Alter- 
nating Current segment of the Air Force electronics 
course, used in the Utah study, plus the Direct Current 
segment of the same course — both modified (and improved) 
by the summer workshop group mentioned above. 

* Teacher reaction to the acceptance of the AC-DC 
teaching package, as presented in the Guide, and modified 
to meet individual teacher requirements, is very positive, 
according to Utah state vocational authorities. 
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* Teachers in selected schools are classroom- 
testing the AC segment of the electronics course, and 
are preparing to test for classroom use other criterion- 
referenced Air Force courses, including solid state 
devices and computer programming, as requested by state 
authorities* 



* * * 



The above is reported, not merely to review the type 
of teacher activity generated by the Utah study, but to 
place into sharper focus a movement from conventional to 
criterion-referenced instruction among teachers who, when 
first exposed, greeted the latter with neutral or skeptical 
attitudes . 

The question arises: Are Utah teachers acceptable, 
models for a movement toward "learning outcome" instruction? 

First off, as noted in the basic report, there is 
every reason to believe that they are a representative 
group of vocational-technical teachers. At the same 
time, it must be stressed that the Utah teaching community 
had vigorous administrative support for the study and the 
follow-on activity — from the state superintendent of 
schools on down — plus strong leadership at the state 
vocational level. Such support was essential to the 
success of the project — and will be, as noted earlier, 
in any similar move away from conventional classroom 
practice. 

In turn, the administrators were motivated by manpower 
studies which revealed a need for change in the Utah 
vocational curriculum — if Utah was to keep pace with 
technological advances related to industrial employment. 
State government, therefore, was motivated by the need 
to retain industry in Utah and attract new industry to 
Utah — as each year thousands of its skilled graduates 
leave the state for lack of desired employment opportunities. 



* * * 



While motivational factors bearing on this issue 
vary from state to state, and locality to locality, it 
can be concluded, based on evidence available, that 
educational reform keyed to "learning output" instruction 
would face, generally, negative teacher attitudes. 

The current system, as noted earlier, offers few 
incentives, if any, for changing the accepted way of 
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doing things. Teaching, as currently taught and prac- 
ticed, usually moves from theory to application of a 
subject, no matter what. Too often it matters little 
whether the student knows all or part of what the teacher 
intends to teach. Too often, if the student lacks 
prerequisite knowledge with which- to understand the 
material being presented, remedial instruction is the 
only solution offered. Too often the attitude prevails: 
"I know my subject; therefore, I know what and how to 
teach. 

In this operational climate, who needs objectives? 
So it goes, too often. 

There ,are a number of widespread beliefs that 
militate against the easy adoption of the notion of 
objectives, of the idea that "I ought to know what I 
intend to achieve before spending public money trying 
to get there." 

One of these is the belief that the teacher's 
function is not that of changing students in some desired 
manner, but rather to "actualize" the student. Actualizing 
means doing things to the student and expecting him to 
react in terms of what was inside him all the^ time. It 
is to deny the responsibility of the change agent; it 
is to deny the responsibility of making sound decisions 
about how the life of another will be influenced. With 
such a belief the need for objectives is nonexistent. 

Another attitude suggests that the teacher believes 
that teaching is the same as assigning. The teacher is in 
charge, and makes assignments. And when he says, "I 'm 
going to hold you responsible for so-and-so" he is, in 
effect, saying, "I'm not going to teach you this, but 
I *m going to expect you to know itfl»" Such attitudes 
about the functions of the instructor again negate the 
importance of objectives. Why go to the trouble of 
defining intended outcomes when it is possible to tell 
the student what you want by pointing to the right 
chapters in the book? 

Perhaps the most problematical attitude is the one 
that says teaching is evaluated best by looking at the 
teacher rather than at the performance of the student. 
Too often, the "good" teacher is determined by what he 
does rather than by what he achieves, rather than by 
what he accomplishes. Too often, the "good" teacher 
is one who holds his chalk properly, maintains discipline, 
speaks loudly enough to be heard, and so on. 
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Under such circumstances it is difficult for most 
teachers to see why objectives are needed. Why take the 
trouble to describe what you want to achieve when you 
already know what to teach and when you are doing a 
good job in the classroom? 

Investigators close to the problem seem to be in 
agreement that the most difficult thing to accomplish 
with teachers is that of modifying their outlook from 
one that is procedure-oriented to one that is outcome- 
oriented. Thus, teachers generally reveal a great 
reluctance to accept responsibility for the outcome of 
their efforts. On the one hand is the belief that the 
teacher teaches, and, on the other hand, the contradictory 
attitude that the teacher really can't be held responsible 
for achievement, or lack of it, on the part of the student. 

If this appraisal is even partially realistic, it 
becomes discouraging to contemplate a movement for 
widespread reform based on "learning output" instruction. 
Yet, here and there — coming out of the woodwork, so to 
speak — are growing numbers of teachers and administrators 
who seek change in the classroom, who are motivated toward 
student-centered instruction. It is these people who 
must be encouraged. And they need proper tools. 

In searching for answers we can at least conclude: 

1. Move ahead faster with criterion-referenced 
instruction in vocational skill areas, where measurement 
and validation of results — because of the "hands on" 
aspects — is more readily obtainable. 

2, Make full use of existing instructional programs, 
such as those offered by the Air Force, which provide 
sets of objectives, already validated, thus eliminating 
the "objective-writing barrier," and replacing invention 
with adaptation. 

Certainly no further research is needed to determine 
whether — in the vocational area, at least — criterion- 
referenced instruction is more effective than conventional 
instruction. And, by their nature, the instructional 
materials involved tend to minimize any negative impact 
an instructor might place on them in the introductory 
phase. Further, literature on the use of criterion- 
referenced materials is readily available to teachers, 
and is accumulating rather rapidly. However, the gap is 
wide between theory and practice — between this literature 
and classroom experience. 
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This leaves one element that deserves further 
attention: namely, more explicit, step-by-step instruc 
tions for the teacher who is dealing for the first time 
with criterion-referenced materials. Such instructions 
should be so explicit that he will be able to tell when 
he is not proceeding according to design. Much basic 



materiaT^exists to accomplish this purpose. The 
instructor schools of the Air Force are a prime source. 
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X. RECOMMENDATION 



Recommendation : That practical guidelines for teachers 
be formulated to implement criterion-referenced courses 
in vocational-skill areas, drawing fully on procedures 
developed and collected by the V. S* Air Force, and on 
other sources — such guidelines to provide step-by-step 
instructions for classroom use* 
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XII • ABSTRACT 



This report supplements Final Report (Project No. 
8-0301, USOE) for the study entitled "The Evaluation of 
Three U, S, Air Force Instructional Systems Within 
Civilian Education/' During that study it became evident 
that the Air Force materials were not being used under 
optimal conditions, particularly the relationship of 
specific performance objectives, and this gave rise to 
this supplemental report, 

Basic guidelines for introducing criterion-referenced 
materials as represented by those of the Air Force, into 
the civilian classroom. Guidelines based on experience 
gained in basic research project, including interviews 
with administrators and instructors, and on other related 
experiences. Report first describes Air Force criterion- 
referenced materials, compares and contrasts them with 
conventional instruction materials. It then offers 
suggestions for selection, preparation for use and use of 
criterion-referenced materials. Teacher attitudes and 
orientation are also covered. 

Report recommends that practical guidelines for 
teachers be formulated to implement criterion-referenced 
courses in vocational skill areas, drawing fully on 
procedures developed and collected by the U. S. Air Force, 
and on other sources. 
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